Tag
#Technical seo
4 posts tagged Technical seo.
Why Google reports 404s for URLs you never created
Search Console flags a /cdn-cgi/l/email-protection 404 you never built. It's Cloudflare rewriting your emails. Here's the cause and the one-line fix.
Jun 16, 2026 · 3 min · Crawlmind Engineering
Your robots.txt change isn't live yet
You edited robots.txt, deployed, and crawlers still see the old rules. robots.txt is edge-cached, often for hours. Here's how to confirm and force it live.
Jun 16, 2026 · 3 min · Crawlmind Engineering
Why your JS-rendered pages can be invisible to AI crawlers
A page can render perfectly in a browser, pass every on-page check, and still be silently de-indexed by an HTTP header your crawler never saw. Here's the X-Robots-Tag trap, why JS-rendered crawls are especially exposed to it, and how to test for it in two minutes.
Jun 15, 2026 · 4 min · Crawlmind Engineering
What "partial" AI-bot access in your robots.txt actually means
Run an AI-crawler check and you'll often see every bot marked "partial" instead of a reassuring green "allowed." That usually means your robots.txt is correct, not broken. Here's how the verdict is computed and when partial is actually a problem.
Jun 15, 2026 · 3 min · Crawlmind Engineering